evals: model-eval suite v1 — design doc, harness, 5 validated seed tasks#756
Draft
stphung wants to merge 2 commits into
Draft
evals: model-eval suite v1 — design doc, harness, 5 validated seed tasks#756stphung wants to merge 2 commits into
stphung wants to merge 2 commits into
Conversation
Captures the interactively-decided design for an in-repo evals/ suite: model-only diff-out tasks mined from git history across four domains (bugfix, feature, UI snapshot, balance/simulator), deterministic graders with an LLM judge on top, fast (~15 min) and slow (simulator) tiers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxnJXgu63wex2CCtfuPy7G
Implements the eval-suite design (docs/explorations/2026-07-15-model-eval-suite.md): - evals/harness/: stdlib-only Python runner. run.py materializes each task in a scratch worktree (bug.patch on the pinned base commit), prompts the model (files in, one unified diff out), enforces per-task edit allowlists + a global denylist (no tests, no .snap re-bless, no harness edits), grades with the repo's own tests/snapshots/simulators, and reports an apply_error -> illegal_edit -> build_error -> test_fail -> pass funnel. validate mode proves every task honest: graders red on the bug state, green after reference.patch. judge.py adds rubric quality scores on top; mine.py lists/extracts task skeletons from git history. - evals/tasks/: 5 hand-authored seed tasks, all red/green-validated — 2 bugfix (damage-percent stacking, serde save-wipe), 1 ui (prestige gauge vs snapshots), 1 feature (shift-hotkey via input replay), and 1 slow-tier balance task graded by the progression simulator. - make eval-validate target; evals/results/ + __pycache__/ gitignored; CLAUDE.md pointer and design-doc history entry. Reference-mode end-to-end run passes 4/4 on the fast tier; the balance task validates red/green against the release simulator. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BxnJXgu63wex2CCtfuPy7G
stphung
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a model-performance eval suite for Quest: the design exploration (
docs/explorations/2026-07-15-model-eval-suite.md) and its v1 implementation underevals/.What it measures: how well a model works on Quest via scoped "files in, diff out" tasks — the model gets a symptom description + selected source files + failing check output, returns one unified diff, and the harness grades it with the repo's own graders (tests, snapshot tests, seeded simulators).
Harness (
evals/harness/, stdlib-only Python)run.py— materializes each task in a scratch worktree (bug.patchapplied to the pinnedeval-basecommit), prompts the model, enforces per-task edit allowlists plus a global denylist (a diff can never touch tests,.snapfiles, build files, or the harness), grades, and reports a diagnostic funnel:apply_error → illegal_edit → build_error → test_fail → pass.--referencedry-runs the whole pipeline with the reference solutions (no API key);validateproves every task honest (graders red on the bug state, green afterreference.patch).judge.py— rubric-based quality scores (convention / minimality / solution_match) on top of the deterministic outcome, never blended into pass rate.mine.py— mines real merged PRs into task skeletons (filters audit-log noise, classifies domain by path, pre-checks the red leg).Seed tasks (
evals/tasks/, all red/green-validated)qb-001-armory-stackingcombat_tests(exact 5x ratio test)qb-002-save-wipeserde(default)vs the frozen save-compat corpusqu-001-prestige-gaugeqf-001-shift-hotkeysqbal-001-meadow-wallsimulator --check-progressionVerification performed
run.py validate(fast tier): 4/4 honest — red on bug, green on reference.run.py validate qbal-001-meadow-wall(slow tier): red across all seeds on the bug state (0 kills, death-looped), green after reference, against the release-build progression simulator.run.py run --reference: full end-to-end pipeline passes 4/4 with reports generated.--libfilter matching zero binary-crate tests → vacuous pass), which is exactly the failure mode the validate contract exists for; fixed tocargo test --bin quest input::replay_tests.Also
make eval-validatetarget;evals/results/and__pycache__/gitignored.CLAUDE.mdsection pointing atevals/README.md.No changes to game source —
src/is untouched; task patches apply only inside the harness's scratch worktree at runtime.🤖 Generated with Claude Code
https://claude.ai/code/session_01BxnJXgu63wex2CCtfuPy7G